Head-mounted display device and operating method of the same

ABSTRACT

A head-mounted display device includes an eye tracking sensor configured to obtain eye information by tracking both eyes of a user, a depth sensor configured to obtain depth information about one or more objects, and a processor configured to obtain information about a gaze point based on the eye information, and determine a measurement parameter of the depth sensor based on the information about the gaze point.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims the benefit of U.S. ProvisionalPatent Application No. 62/832,544, filed on Apr. 11, 2019, in the UnitedStates Patent and Trademark Office, and claims priority under 35 U.S.C.§ 119 to Korean Patent Application No. 10-2019-0106133, filed on Aug.28, 2019, in the Korean Intellectual Property Office, the disclosures ofwhich are herein incorporated by reference in their entireties.

BACKGROUND 1. Field

The disclosure relates to a head-mounted display device capable ofdetermining a gaze point in a real space and obtaining depth informationusing parameters optimized for the gaze point, and a method of operatingthe same.

2. Description of Related Art

The real world environment in which we live may be characterized as athree-dimensional (3D) space. A person perceives a 3D space due to astereoscopic effect obtained by combining visual information seen by apair of eyes. However, because a photo or a video taken by a generaldigital device is generated via a technology of expressing 3Dcoordinates in two-dimensional (2D) coordinates, the photo or the videodoes not include information, such as a depth of objects, about space.To express such a sense of space, 3D cameras or display products, whichemploy two cameras together to capture and display stereoscopic images,have been developed.

To express a sense of 3D space, depth information about objects withinthe real space may be necessary. Depth sensing with regard to the depthinformation has been performed on all ranges of space that a depthsensor is capable of measuring, without considering a region of interest(ROI) of a user. In particular, the depth sensor that projects light toperform depth sensing drives an infrared (IR) light-emitting device(LED) to project light in all ranges of space, and thus powerconsumption increases due to the driving of the IR LED. In addition, toobtain depth information about all ranges of space, an amount ofcomputation increases. Accordingly, power consumption also increases.Because the power consumption of the depth sensor increases, there is aproblem of mounting the depth sensor in a small device having limitedpower and/or computational resources.

In addition, conventional depth sensing methods may provide inaccuratedepth sensing due to weakness of the depth sensors.

SUMMARY

Aspects of the disclosure relate to a head-mounted display device thatdetermines a gaze point and obtains depth information about a presetregion of interest (ROI) with respect to the gaze point using parametersoptimized for the gaze point and a method of operating the head-mounteddisplay device.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments of the disclosure.

According to an embodiment of the disclosure, a head-mounted displaydevice may include an eye tracking sensor configured to track a positionof focus of a left eye of a user and a position of focus of a right eyeof the user; a depth sensor configured to obtain depth information aboutone or more objects; and a processor configured to determine a gazepoint based on the position of focus of the left eye of the user and theposition of focus of the right eye of the user, and determine ameasurement parameter of the depth sensor based on the gaze point.

The processor may be further configured to obtain two-dimensional (2D)location information of the gaze point based on the eye information.

The processor may be further configured to obtain estimated depthinformation of the gaze point based on the position of focus of the lefteye of the user and the position of focus of the right eye of the user.

The depth sensor may be further configured to obtain estimated depthinformation of the gaze point based on the position of focus of the lefteye of the user and the position of focus of the right eye of the user.

The measurement parameter of the depth sensor may include at least oneof a parameter with respect to a target region, a parameter with respectto an output of an emission light, or a parameter with respect tosensing of a reflection light.

The processor may be further configured to re-determine the measurementparameter of the depth sensor based on the depth information about theROI, and wherein the depth sensor may be further configured to re-obtainthe depth information about the ROI again according to the measurementparameter.

The depth sensor may be further configured to obtain the depthinformation about the ROI by using at least one of a time of flight(TOF) method, a structured light (SL) method, or a stereo image (SI)method.

When the depth sensor includes a TOF depth sensor, the processor may befurther configured to determine the measurement parameter based on the2D location information of the gaze point such that some light sourcescorresponding to the gaze point among light sources included in thedepth sensor are driven, and wherein the depth sensor may be furtherconfigured to obtain the depth information about the ROI by driving thesome light sources.

The head-mounted display device may further include a display displayinga real space including the ROI, and wherein the processor may be furtherconfigured to control the display to display at least one virtual objecton the ROI based on the depth information about the ROI.

The processor may be further configured to set the measurement parameterof the depth sensor to a first parameter, based on the first parameter,control the depth sensor to obtain whole depth information about a spacethat the depth sensor is capable of sensing, the space including theROI, based on the whole depth information, obtain first depthinformation about the ROI, based on the first depth information, set themeasurement parameter of the depth sensor as a second parameter, and,based on the second parameter, control the depth sensor to obtain seconddepth information about the ROI.

According to another embodiment of the disclosure, a method of operatingan electronic device may include tracking a position of focus of a lefteye of a user and a position of focus of a right eye of the user;obtaining a gaze point based on the position of focus of the left eye ofthe user and the position of focus of the right eye of the user; anddetermining a measurement parameter of a depth sensor based on the gazepoint.

According to another embodiment of the disclosure, one or morenon-transitory computer-readable recording media may have recordedthereon a program for controlling an apparatus to execute the methodsherein.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a diagram illustrating an electronic device according to anembodiment of the disclosure;

FIGS. 2, 3A, 3B, 3C, and 3D are diagrams for describing a methodperformed by an electronic device to track the eye of a user accordingto an embodiment of the disclosure;

FIGS. 4A and 4B are diagrams for describing a method performed by anelectronic device to obtain depth information about a gaze pointaccording to an embodiment of the disclosure;

FIGS. 5A and 5B are diagrams for describing a method performed by anelectronic device to determine measurement parameters of a depth sensorbased on a gaze point of a user according to an embodiment of thedisclosure;

FIGS. 6, 7A, and 7B are reference diagrams for describing a methodperformed by an electronic device to determine measurement parameters inthe case of a depth sensor using a time of flight (TOF) method accordingto an embodiment of the disclosure;

FIGS. 8 and 9 are reference diagrams for describing a method performedby an electronic device to determine measurement parameters in the caseof a depth sensor using a structured light (SL) method according to anembodiment of the disclosure;

FIG. 10 is a reference diagram for describing a method performed by anelectronic device to determine measurement parameters in the case of adepth sensor uses a stereo image (SI) method according to an embodimentof the disclosure;

FIG. 11 is a diagram for describing a method performed by an electronicdevice to display a virtual object according to an embodiment of thedisclosure;

FIG. 12 is a flowchart of a method of operating an electronic device,according to an embodiment of the disclosure;

FIG. 13 is a diagram for describing a method performed by an electronicdevice to obtain depth information according to an embodiment of thedisclosure;

FIG. 14 is a flowchart illustrating a method performed by an electronicdevice to obtain depth information according to an embodiment of thedisclosure;

FIG. 15 is a diagram illustrating an example in which an electronicdevice repeatedly performs operations of obtaining depth information ofFIG. 14 according to an embodiment of the disclosure;

FIG. 16 is a diagram illustrating an example in which an electronicdevice provides a virtual object using an augmented reality (AR) methodaccording to an embodiment of the disclosure;

FIG. 17 is a diagram illustrating an example in which an electronicdevice recognizes a face of a person using depth information accordingto an embodiment of the disclosure;

FIG. 18 is a block diagram illustrating a configuration of an electronicdevice according to an embodiment of the disclosure;

FIG. 19 is a block diagram illustrating a configuration of an electronicdevice according to another embodiment of the disclosure;

FIGS. 20 and 21 are diagrams for describing a method performed by anelectronic device to automatically adjust a focus according to anembodiment of the disclosure; and

FIG. 22 is a diagram for describing a method performed by an electronicdevice to perform eye based spatial modeling according to an embodimentof the disclosure.

DETAILED DESCRIPTION

Terms used herein will be described, and the disclosure will bedescribed in detail.

Although terms used in the disclosure are selected with general termspopularly used at present under the consideration of functions in thedisclosure, the terms may vary according to the intention of those ofordinary skill in the art or introduction of new technology. Inaddition, in a specific case, terms may be selected and the meaning ofthe terms may be disclosed in a corresponding description of thedisclosure. Thus, the terms used in the disclosure should be defined notby the simple names of the terms, but by the meaning of the terms andthe contents throughout the disclosure.

Throughout the entirety of the disclosure, a certain part may be assumedto include a certain component, and the term ‘including’ means that acorresponding component may further include other components unless aspecific meaning opposed to the corresponding component is described.The term used in the embodiments such as “unit” or “module” indicate aunit for processing at least one function or operation, and may beimplemented in hardware, software, or in a combination of hardware andsoftware.

Throughout the disclosure, the expression “at least one of a, b or c”indicates only a, only b, only c, both a and b, both a and c, both b andc, all of a, b, and c, or variations thereof.

Hereinafter, embodiments of the disclosure will be described in detailwith reference to the attached drawings to allow those of ordinary skillin the art to easily carry out the embodiments. However, the disclosuremay be implemented in various forms, and the embodiments are not limitedto the embodiments described herein. To clearly describe the disclosure,parts that are not associated with the description have been omittedfrom the drawings, and throughout the specification, identical referencenumerals refer to identical parts.

FIG. 1 is a diagram illustrating an electronic device 100 according toan embodiment of the disclosure.

Referring to FIG. 1, the electronic device 100 according to anembodiment of the disclosure may be a glasses type wearable device. Theglasses type wearable device may be implemented as a head-mounteddisplay (HMD) that is mountable on a head. For example, the HMD mayinclude a device in the form of glasses, a helmet, a hat, and the like,but the type and form of the electronic device 100 are not limitedthereto.

In addition, the electronic device 100 according to an embodiment of thedisclosure may be implemented in various electronic devices such as amobile phone, a smart phone, a laptop computer, a desktop, a tablet PC,an e-book terminal, a digital broadcasting terminal, a personal digitalassistant (PDA), a portable multimedia player (PMP), a navigationdevice, an MP3 player, a camcorder, an Internet protocol television(IPTV), a digital television (DTV), a wearable device, etc., but thetype and form of the electronic device 100 are not limited thereto.

In an embodiment of the disclosure, the term “user” refers to a personwho controls functions or operations of the electronic device 100, andmay include an administrator or an installation engineer.

The electronic device 100 according to an embodiment of the disclosuremay be a device that provides at least one virtual object in the form ofaugmented reality (AR), mixed reality (MR), or virtual reality (VR).

When providing the virtual object in the form of AR or MR, theelectronic device 100 may display the virtual object on a display suchthat the virtual object matches the shape, arrangement, distance, anddepth of a real object in the real world. For example, the electronicdevice 100 may overlap and display an image of the virtual object on thereality of the real world, but the display of images is not limitedthereto.

The electronic device 100 according to an embodiment of the disclosuremay include a depth sensor 150.

The depth sensor 150 may obtain depth information about one or moreobjects included in the real world. The depth information may correspondto a distance from the depth sensor 150 to a specific object. As thedistance from the depth sensor 150 to the specific object increases, adepth value may correspondingly increase.

As shown in FIG. 1, on a three-dimensional (3D) space, the X axis may bea reference axis passing left and right across the electronic device100, the Y axis may be a reference axis passing up and down across theelectronic device 100, and the Z axis may be a reference axis passingforward and backward across the electronic device 100. The X axis, Yaxis and Z axis may be perpendicular to each other. However, any of the(+/−) X axis, the (+/−) Y axis, and the (+/−) Z axis may be assigned toa particular direction.

Accordingly, the depth information according to an embodiment of thedisclosure may mean a distance on the Z axis from the depth sensor 150to the specific object.

The depth sensor 150 according to an embodiment of the disclosure mayobtain depth information of an object in various ways. For example, thedepth sensor 150 may obtain the depth information using at least one ofa time of flight (TOF) method, a structured light (SL) method, or astereo image (SI) method. Each method will be described later in detail.

The depth sensor 150 according to an embodiment of the disclosure mayinclude at least one camera (an image sensor). The depth sensor 150 mayobtain depth information about an actual space included in a field ofview (FOV) of the camera. Hereinafter, the actual space of the rangethat is capable of being sensed by the depth sensor 150 is referred toas a ‘whole space’. In general, the ‘whole space’ corresponds to a wholeregion of detection of the image sensor.

Meanwhile, when a user of the electronic device 100 gazes or focuses ata partial space or a particular region in the whole space, depthinformation about the remaining space excluding the partial space may beless important than the space at which the user gazes. For example,while the user of the electronic device 100 gazes at a space 50 around adesk in a room space (a whole space 10) shown in FIG. 1, the electronicdevice 100 obtains depth information about the whole space 10 using thedepth sensor 150. The depth information about the remaining spaceexcluding the space 50 around the desk may be information of a lowerimportance than that of depth information of the space 50 around thedesk. In the whole space 10, a point at which the user gazes and thesurrounding region of the point at which the user gazes are referred toas a region of interest (ROI). According to an embodiment of thedisclosure, the ROI may be a predetermined region with respect to thepoint within the whole space at which the user gazes.

In addition, when the electronic device 100 obtains the depthinformation about the whole space 10, because an amount of computationincreases, power consumption may correspondingly increase. As a result,the response speed of the electronic device 100 may decrease. Inaddition, when the electronic device 100 does not obtain the depthinformation by targeting only a partial space (e.g., the gaze point ofthe user of the electronic device 100), but obtains the depthinformation by targeting the whole space 10, the accuracy of the depthinformation about the gaze point of the user of the electronic device100 may be lowered.

Accordingly, the electronic device 100 according to an embodiment of thedisclosure may determine the point at which the user gazes in the wholespace 10. For example, the electronic device 100 may track a gaze ofeach eye of the user, obtain eye information thereof, and determine thepoint (the gaze point) at which the user gazes based on the eyeinformation.

The electronic device 100 according to an embodiment of the disclosuremay adjust a measurement parameter of the depth sensor 150 based on thedetermined gaze point, thereby reducing power consumption required forobtaining the depth information and improving the accuracy of the depthinformation.

The measurement parameters of the depth sensor 150 may be numericalinformation +set in advance in the depth sensor 150 as parametersnecessary to obtain depth information when obtaining the depthinformation about at least one object using the depth sensor 150. Forexample, the measurement parameters of the depth sensor 150 may includea parameter with respect to a target region for emitting light, aparameter with respect to a reflection light sensing region for sensingthe reflection light, a parameter with respect to an output pattern ofthe emission light, a parameter with respect to an output size of theemission light, a parameter with respect to a sensing speed for sensingthe reflection light, a sensing period, or sensing sensitivity, etc.

FIGS. 2, 3A, 3B, 3C, and 3D are diagrams for describing a method,performed by the electronic device 100, of tracking the eye of a useraccording to an embodiment of the disclosure.

Referring to FIG. 2, the electronic device 100 may track the eye of theuser. In general, the direction of the ‘eye’ refers to a direction thatthe user views, and ‘eye tracking’ refers to a process of measuring theuser's eye (e.g., a point 210 at which a user gazes) and may beperformed by tracking positions and movement of both eyes.

The electronic device 100 according to an embodiment of the disclosuremay include an eye tracking sensor 160 to track the eye of the user. Theeye tracking sensor 160 according to an embodiment of the disclosure mayinclude a first eye tracking sensor 161 for tracking the user's left eyeand a second eye tracking sensor 162 for tracking the user's right eye.The first eye tracking sensor 161 and the second eye tracking sensor 162have the same structure and operate in the same manner.

FIG. 3A is a diagram for describing a method of tracking the eye of theuser based on an amount of light reflected from a user's eye 320.

The first eye tracking sensor 161 and the second eye tracking sensor 162according to an embodiment of the disclosure have the same structure andoperate in the same manner, and thus the first eye tracking sensor 161will be described in FIG. 3A.

Referring to FIG. 3A, the first eye tracking sensor 161 according to anembodiment of the disclosure may include an illuminator 301 thatprovides light to the user's eye 320 and a detector 302 that detectslight. The illuminator 301 may include a light source that provideslight and a scanning mirror that controls a direction of the lightprovided from the light source. The scanning mirror may control thedirection to direct the light provided from the light source toward theuser's eye 320 (e.g., a cornea 310). The scanning mirror may include astructure by which a reflection angle may mechanically change to reflectthe light provided from the light source and direct the light toward theuser's eye 320, and may scan a region including the cornea 310 using thelight provided from the light source according to the changed reflectionangle.

The detector 302 may detect the light reflected from the user's eye 320and measure an amount of the detected light. For example, when the lightis reflected from the center of the user's cornea 310, the amount of thelight detected by the detector 302 may be maximum. Accordingly, when theamount of the light detected by the detector 302 is maximum, the firsteye tracking sensor 161 may determine an eye direction 340 of the user'seye 320 based on a point 330 at which the light is incident on andreflected from the user's eye 320. For example, when the amount of thelight is maximum, the first eye tracking sensor 161 may determine thedirection 340 connecting the point 330 at which the light is incident onand reflected from the user's eye 320 and a center point of the of theeye of the user's eye 320 (e.g. the user's left eye), but the method isnot limited thereto.

In addition, the second eye tracking sensor 162 may also determine theeye direction of a user's eye (e.g., the user's right eye) in the samemanner as described with reference to FIG. 3A.

FIG. 3B is a diagram for describing a method of tracking the eye of auser based on a position of a reflection light reflected from a user'seye.

The first eye tracking sensor 161 and the second eye tracking sensor 162according to an embodiment of the disclosure have the same structure andoperate in the same manner, and thus the first eye tracking sensor 161will be described in FIG. 3B. The first eye tracking sensor 161 mayinclude an illuminator 351 and a capturer 352. The illuminator 351 mayinclude an infrared light emitting diode (IR LED). As shown in FIG. 3B,the illuminator 351 may include a plurality of light emitting diodesdisposed at different positions. The illuminator 351 may provide light(e.g., an infrared light) to the user's eye when the user's eye istracked. Because the light is provided to the user's eye, the reflectionlight may be generated in the user's eye.

The capturer 352 may include at least one camera. At this time, the atleast one camera may include an infrared camera IR. The electronicdevice 100 may track the user's eye (e.g., the user's left eye) using animage of the user's eye captured by the capturer 352. For example, thefirst eye tracking sensor 161 may track the eye of the user by detectingthe pupil and the reflection light from the image of the user's eye. Thefirst eye tracking sensor 161 may detect the positions of the pupil andthe reflection light from the image of the user's eye and determine theeye direction of the user's eye based on the relationship between theposition of the pupil and the position of the reflection light.

For example, the first eye tracking sensor 161 may detect a pupil 370and a reflection light 381 from a captured first eye image 361 anddetermine an eye direction 391 of the user's eye based on therelationship between the position of the pupil 370 and the position ofthe reflection light 381. In the same manner, the first eye trackingsensor 161 may detect the pupil 370 and reflection lights 382, 383, 384,and 385, respectively, from second to fifth eye images 362, 363, 364,and 365 and determine eye directions 392, 393, 394, and 395 of theuser's eye based on the relationship between the position of the pupil370 and the positions of the reflection lights 382, 383, 384, and 385respectively.

In addition, the second eye tracking sensor 162 may also determine theeye direction of the user's eye (e.g., the user's right eye) in the samemanner as described with reference to FIG. 3B.

FIG. 3C is a diagram illustrating a three-dimensional (3D) eye model ofthe eye of a user.

Referring to FIGS. 2 and 3C, the electronic device 100 may determine theeye direction of a user's left eye using the first eye tracking sensor161 and determine the eye direction of a user's right eye using thesecond eye tracking sensor 162. For example, the electronic device 100may determine the eye direction based on an average eye model of ahuman. The eye model may be modeled by assuming that a human's eye 3100is in a spherical shape and that the eye ideally rotates according tothe eye direction. In addition, the eyeball model may be expressedmathematically as shown in Equations 1 and 2 below.

x=d·tan α,

y=d·sec α·tan β  [Equation 1]

β=sin⁻¹(dif f_y/r

α=sin⁻¹(dif f_x/r cos β)   [Equation 2]

Equation 1, d denotes a distance between a center 3150 of the user's eyeand a virtual screen 3200, α denotes an angle at which the user's eyerotates in an x axis direction based on the case where the user's eyesgazes at the front of the virtual screen 3200, and β denotes an angle atwhich the user's eye rotates in a y axis direction based on the casewhere the user's eyes gazes at the front of the virtual screen 3200.Also, in Equation 2, r denotes the radius of a sphere when assuming thatthe user's eye is the sphere.

According to an embodiment of the disclosure, the first eye trackingsensor 161 may measure the degree of rotation (e.g., α and β) of theuser's eye (e.g., the left eye) using the method described withreference to FIGS. 3A and 3B. The electronic device 100 may calculatetwo-dimensional (2D) coordinates of the eye direction of the user's eyeon the virtual screen 3200 using the degree of rotation (α and β) of theuser's eye.

FIG. 3D is a reference diagram for describing a method of performingcalibration of the eye tracking sensor 160 according to an embodiment ofthe disclosure.

As an example, when the user first uses the electronic device 100, theelectronic device 100 may calibrate the first eye tracking sensor 161and the second eye tracking sensor 162 to accurately measure the user'sleft and right eyes. The electronic device 100 may output virtual imagesVI1, VI2, and VI3 of different depths (e.g., d1, d2, and d3) on which aplurality of points (generally 9) for inducing the user's eye and inducea point at which the user gazes with respect to each of the plurality ofpoints.

When the user gazes at the point included in each of the virtual imagesVI1, VI2, and VI3, the electronic device 100 may store information (eyeinformation) output from the eye tracking sensor 160 in the form of atable, array, or any other data storage mechanism.

As described with reference to FIG. 3A, the electronic device 100 maystore information of the reflection angle of the scanning mirror and anamount of light as the eye information for each point in a method usingthe amount of light reflected from the user's cornea and may store animage including the user's eye and the reflection light captured at eachpoint as the eye information in a method capturing the user's eye byusing an infrared light.

The electronic device 100 may determine the eye direction of the user'seye by comparing the stored eye information with the measured eyeinformation output from the eye tracking sensor 160. The electronicdevice 100 may determine the eye direction of the user's left eye usingthe eye information output from the first eye tracking sensor 161 andmay determine the eye direction of the user's right eye using the eyeinformation output from the second eye tracking sensor 162.

The electronic device 100 may use the eye direction of the user's lefteye, the eye direction of the right eye, and the distance between botheyes, as illustrated in FIG. 2, to estimate the coordinates of the point210 at which the user gazes in the whole space 10.

For example, the electronic device 100 may set the point 210 at whichthe user gazes to be mapped as 2D coordinate information (e.g., xcoordinate value and y coordinate value) in the whole space 10 describedwith reference to FIG. 1 by using coordinate mapping, etc. or may storethe point 210 in the form of the table.

FIGS. 4A and 4B are diagrams for describing a method, performed by theelectronic device 100, of obtaining depth information about a gaze point430 according to an embodiment of the disclosure.

Referring to FIG. 4A, the electronic device 100 may use the vergence ofthe eye direction of the right eye and the eye direction of the left eye(intersecting two virtual straight lines indicating an eye direction) toestimate the depth information about a point at which a user gazes.

For example, as shown in FIG. 4B, the electronic device 100 maycalculate a distance value Z1 to the gaze point 430 (a point at whichthe eye of both eyes verge) based on a first eye direction 410corresponding to the left eye, a second eye direction 420 correspondingto the right eye, and the distance between both eyes. The electronicdevice 100 may obtain the depth information about the gaze point 430 byusing eye information of both eyes measured using the eye trackingsensor 160 and Equation 3 below according to the geometrical arrangementillustrated in FIG. 4B.

$\begin{matrix}\begin{matrix}{\frac{- z}{\Delta \; x} = {\left. \frac{D - z}{a}\Rightarrow z \right. = {\frac{\Delta \; {xD}}{{\Delta \; x} - a}.}}}\end{matrix} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack\end{matrix}$

In Equation 3, Δx denotes a difference between an x coordinate x1 of theleft eye and an x coordinate x2 of the right eye on a virtual screen450. In this regard, the difference may be calculated assuming that they coordinate of the left eye and the y coordinate of the right eye arethe same. Also, in Equation 3, a denotes the distance between the user'sboth eyes, and a preset value (e.g., 7 cm) may be used. Also, D denotesa distance between the user's eyes and the virtual screen 450.

The electronic device 100 may obtain the distance value Z1 to the gazepoint 430 at which the eyes of user's both eyes verge as the sum of a zvalue and a D value.

Alternatively, the electronic device 100 may estimate the depthinformation (e.g., Z1) about the gaze point 430 based on an angle formedbetween the first eye direction 410 and the second eye direction 420.The smaller the angle formed between the first eye direction 410 and thesecond eye direction 420 is, the greater the distance to the gaze point430, and the greater the angle formed between the first eye direction410 and the second eye direction 420 is, the closer the gaze point 430.

FIGS. 5A and 5B are diagrams for describing a method, performed by theelectronic device 100, of determining measurement parameters of thedepth sensor 150 based on a gaze point of a user according to anembodiment of the disclosure.

The measurement parameters of the depth sensor 150 may be numericalinformation that needs to be set in advance in the depth sensor 150 asparameters necessary to obtain depth information when obtaining thedepth information about at least one object using the depth sensor 150.For example, the measurement parameters of the depth sensor 150 mayinclude a parameter with respect to a target region for emitting light,a parameter with respect to a reflection light sensing region forsensing the reflection light, a parameter with respect to an outputpattern of the emission light, a parameter with respect to an outputsize of the emission light, a parameter with respect to a sensing speedfor sensing the reflection light, a sensing period, or sensingsensitivity, etc.

Referring to FIGS. 5A and 5B, an electronic device 100 according to anembodiment of the disclosure may include the eye tracking sensor 160, aprocessor 120, and the depth sensor 150.

Referring to FIG. 5A, the eye tracking sensor 160 according to anembodiment of the disclosure may obtain the eye information of a user.This is described in detail with reference to FIGS. 2 to 3D, and thus adetailed description thereof is omitted.

The eye tracking sensor 160 may transmit the eye information of the userto the processor 120. The processor 120 may obtain information about agaze point based on the eye information of the user.

For example, the processor 120 may obtain 2D location information of thegaze point based on the eye direction (a first eye direction) for theuser's left eye and the eye direction (a second eye direction) for theuser's right eye included in the eye information. The processor 120 mayuse the first eye direction and the second eye direction to determine 2Dcoordinates (e.g., x coordinate values and y coordinate values) in awhole space with respect to the gaze point of the user.

The processor 120 may determine the measurement parameters of the depthsensor 150 by using the 2D location information of the gaze point. Forexample, the processor 120 may determine a parameter of a target regionby using the 2D location information of the gaze point. The processor120 may determine a predetermined ROI as the target region with respectto the gaze point.

The depth sensor 150 may obtain the depth information based on thedetermined parameter of the target region. For example, when emittinglight, a light emitter 170 may limit a light emitting region to thetarget region (a region corresponding to the gaze point) or when sensinga reflection light, a sensor 180 may limit the light emitting region asthe target region (a region corresponding to the gaze point).Accordingly, the depth sensor 150 may obtain depth information aboutobjects included in the target region (the gaze point and thesurrounding region of the gaze point) other than the whole space.

Referring to FIG. 5B, the eye tracking sensor 160 according to anembodiment of the disclosure may obtain the eye information of the userand transmit the eye information to the processor 120. The processor 120may obtain estimated depth information of the gaze point based on theeye information of the user. For example, the processor 120 may obtainthe estimated depth information (e.g., z coordinate values) of the gazepoint based on the first eye direction and the second eye direction ofthe user included in the eye information.

The processor 120 according to an embodiment of the disclosure may usethe estimated depth information of the gaze point to determine theparameter with respect to the output of the emission light (e.g., theoutput pattern of the emission light and the magnitude of the output ofthe emission light) of the depth sensor 150.

For example, the processor 120 may determine the parameter of the depthsensor 150 such that the output pattern (the width of a light pulse) ofthe light emitted from the light emitter 170 is reduced when anestimated depth is small (when the distance is near) and the outputpattern (the width of the light pulse) of the light emitted from thelight emitter 170 increases when the estimated depth is large (when thedistance is far). In addition, when the sensor 180 senses the reflectionlight, the processor 120 may determine the parameter of the depth sensor150 such that when the estimated depth is small, the sensing speed orthe sensing period increases. Conversely, when the estimated depth islarge, the sensing speed or the sensing period decreases. Accordingly,the depth sensor 150 may obtain depth information of objects included inthe ROI based on the determined parameters with respect to the outputpattern (the width of a light pulse) of the light and the sensing speedor the sensing period. At this time, the ROI may be a preset region withrespect to the gaze point.

In addition, the processor 120 may determine the parameter of the depthsensor 150 such that when the estimated depth of the gaze point issmall, the output of the light emitted from the light emitter 170 isreduced. Conversely, when the estimated depth is large, the output ofthe light emitted from the light emitter 170 increases. In addition,when the sensor 180 senses the reflection light, the electronic device100 may determine the parameter of the depth sensor 150 such that whenthe estimated depth is small, sensitivity is reduced. Conversely, whenthe estimated depth is large, the sensitivity increases. Accordingly,the depth sensor 150 may obtain depth information of objects included inthe ROI in consideration of the determined magnitude of the output oflight and the sensing sensitivity. At this time, the ROI may be a presetregion with respect to the gaze point.

Hereinafter, examples in which measurement parameters of a depth sensorare determined based on the information (the 2D location information ofthe gaze point and the estimated depth information of the gaze point)about the gaze point will be described in detail according to the typeof the depth sensor.

FIGS. 6 to 7B are reference diagrams for describing a method, performedby an electronic device, of determining measurement parameters when thedepth sensor 150 uses a TOF method according to an embodiment of thedisclosure.

The depth sensor 150 according to an embodiment of the disclosure mayinclude a time-of-flight (TOF) depth sensor 610.

The TOF method is a method of analyzing a time taken for light to bereflected from an object 620 and return and measuring a distance d tothe object 620. The TOF depth sensor 610 may include a light emitter 630and a sensor unit 640.

The light emitter 630 may be arranged to surround the sensor unit 640,but the configuration is not limited thereto. The sensor 640 may bearranged to surround the light emitter 630.

The light emitter 630 may include a light source that generates light ofa predetermined wavelength. The light source may include an infraredlight emitting diode (IR LED) a laser diode (LD) capable of emittinglight of an infrared wavelength invisible to the human eye, but is notlimited thereto, and the wavelength band and the kind of the lightsource may be variously configured. The light emitter 630 may emit thelight to the object 620 by driving the light source according to acontrol signal. For example, the light emitter 630 may emit the light tothe object 620 by repeatedly turning on and turning off the light sourcelight in an alternating fashion.

The sensor unit 640 may sense a reflection light that is reflected fromthe object 620 and returns. For example, the sensor unit 640 may includean optical sensing element such as a pinned photo diode (PPD), aphotogate, a charge coupled device (CCD), etc. The sensor unit 640 mayinclude a plurality of sensors arranged in an array, and one cell 650included in the array may be configured to form a pair of an in-phasereceptor 651 and an out-phase receptor 652. At this time, the in-phasereceptor 651 may be activated only in an in-phase state (while the lightemitter 630 emits light) to detect the light, and the out-phase receptor652 may be activated only in an out-phase state (while the light emitter630 does not emit the light) to detect the light.

FIG. 7A is a graph illustrating light 710 emitted from the light emitter630, a reflection light 720, light received by an in-phase receptor, andlight received at an out-phase receptor.

The light 710 emitted from the light emitter 630 may be reflected fromthe object 620, which is separated from the depth sensor 610 by thepredetermined distance d, and return. The reflection light 720 may causea time delay to occur by the predetermined distance d compared to theemitted light 710. For example, the light may not be received during acertain section in the operation section of the in-phase receptoractivated only when the light emitter 630 emits the light 710. Also, tothe contrary, the reflected light may be received during a certainsection in the operation section of the out-phase receptor. The in-phasereceptor and the out-phase receptor may measure an amount of thereceived light. For example, the in-phase receptor and the out-phasereceptor may receive light reflected from an object, generate andaccumulate electrons, thereby measuring an amount (a charge amount) ofthe accumulated electrons. The distance d to the object 620 may becalculated as in Equation 4 below.

$\begin{matrix}{d = {\frac{c \cdot t}{2}\frac{q\; 2}{{q\; 1} + {q\; 2}}}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

Here, c denotes a light speed, t denotes the length of a pulse of thelight 710, q1 denotes the amount of accumulated charges (the amount ofcharges measured by the in-phase receptor) when the light is emitted,and q2 denotes an amount of accumulated charges (the amount of chargesmeasured by the out-phase receptor) when light is not emitted. That is,the farther the distance d, the later the point to start receiving thereflected light. Accordingly, q2 may relatively increase compared to q1.As a result, the TOF depth sensor 610 may calculate depth informationusing Equation 4.

Meanwhile, the electronic device 100 according to an embodiment of thedisclosure may determine a gaze point in a whole space by using eyeinformation of a user. The electronic device 100 may obtain the eyeinformation of the user using the eye tracking sensor 160 and may obtain2D location information of the gaze point based on the eye informationof the user.

The electronic device 100 according to an embodiment of the disclosuremay control the light emitter 630 and the sensor unit 640 based on the2D location information of the gaze point.

Referring back to FIG. 6, the light emitter 630 according to anembodiment of the disclosure may split into a plurality of regions. Theelectronic device 100 may drive light sources for each region. Theelectronic device 100 may drive light sources included in a regioncorresponding to the 2D location information of the gaze point among theplurality of regions. For example, when the light emitter 630 splitsinto four regions A1, A2, A3, and A4, the electronic device 100 maydrive light sources included in the first region Al corresponding to the2D location information of the gaze point and may not drive lightsources included in the remaining second to fourth regions A2, A3, andA4.

Accordingly, power consumption according to driving of the light sourcemay be reduced. In addition, the sensor unit 640 may also split into aplurality of regions. The electronic device 100 may drive only the lightsources included in the region corresponding to the 2D locationinformation of the gaze point among the plurality of regions. Forexample, when the sensor unit 640 split into four regions B1, B2, B3,and B4, the electronic device 100 may sense a signal by driving sensorsincluded in only the first region B1 corresponding to the 2D locationinformation of the gaze point. Accordingly, power consumption accordingto calculation of the depth information may be reduced.

The electronic device 100 according to an embodiment of the disclosuremay drive only a light source of a region corresponding to the 2Dlocation information of the gaze point in the light emitter 630 includedin the depth sensor, based on the 2D location information of the gazepoint, sense a signal by driving only sensors included in the regioncorresponding to the 2D location information of the gaze point in thesensor unit 640, and calculate the depth information.

In addition, the electronic device 100 according to an embodiment of thedisclosure may determine the length, magnitude, etc. of the pulse of theemitted light 710 as optimized parameters based on the estimated depthinformation of the gaze point. For example, the electronic device 100may determine the length, magnitude (the output of light), etc. of thepulse of the light 710 based on the estimated depth information of thegaze point. For example, when the estimated depth is small (when thedistance is near), the electronic device 100 may reduce the length (thewavelength of the light) of the pulse of the light 710 projected by thelight emitter 630 and reduce the output of the pulse of the light 710.In addition, when the estimated depth is large (i.e., the distance isfar), the electronic device 100 may increase the length (the wavelengthof the light) of the pulse of the light 710 projected by the lightemitter 630 and increase the output of the pulse of the light 710.

In addition, when the sensor unit 640 senses a reflection light pulse,the electronic device 100 may reduce the sensing speed and lower thesensing sensitivity when the estimated depth is small. In addition, theelectronic device 100 may decrease the sensing speed and increase thesensing sensitivity when the estimated depth is large. As describedabove, because the measurement parameters of the depth sensor may bedetermined as parameters optimized for the gaze point, the accuracy ofthe depth information about the gaze point may be improved.

The electronic device 100 may calculate depth information about the ROIwithout calculating depth information of the whole space. Accordingly,the speed of calculating the depth information may increase, and powerconsumption may be reduced.

In addition, the electronic device 100 according to an embodiment of thedisclosure may re-optimize the measurement parameters of the depthsensor based on the depth information (actual depth information)calculated by the depth sensor, and perform depth sensing using there-optimized measurement parameters. Accordingly, the accuracy of thedepth information may be further improved.

FIG. 7B is a diagram for describing a method, performed by theelectronic device 100, of determining measurement parameters of a depthsensor using a matching table 750 according to an embodiment of thedisclosure.

The matching table 750 illustrated in FIG. 7B is a table thatcorresponds to the eye information of the user and the measurementparameters of the depth sensor and may be stored in the electronicdevice 100 according to an embodiment of the disclosure. The electronicdevice 100 according to an embodiment of the disclosure may control thelight emitter 630 and the sensor unit 640 of FIG. 6 using the matchingtable 750.

The electronic device 100 may obtain the 2D location information (xcoordinate and y coordinate) of the gaze point and the estimated depthinformation (z coordinate) based on the eye information of the userobtained from the eye tracking sensor 160, and determine the measurementparameters corresponding to the obtained 2D location information and theestimated depth information by using the matching table 750.

For example, when the 2D location information of the gaze point is (3,4) and the z value of the estimated depth information is 3, theelectronic device 100 may drive light sources included in the firstregion Al of the light emitter 630 and may not drive light sourcesincluded in the remaining second to fourth regions A2, A3, and A4. Inaddition, the electronic device 100 may drive only sensors included inthe first region B1 of the sensor unit 640 to sense a signal. Theelectronic device 100 may control the light emitter 630 such that themagnitude of the pulse of the light 710 output from the light emitter630 is 2, the period of the pulse of the light 710 is 2 ms, and the dutycycle of the pulse of the light 710 is 10%.

Alternatively, when the 2D location information of the gaze point is(−2, −5) and the z value of the estimated depth information is 5, theelectronic device 100 may drive light sources included in the thirdregion A3 of the light emitter 630 and may not drive light sourcesincluded in the remaining first, second, and fourth regions A1, A2, andA4. In addition, the electronic device 100 may drive only sensorsincluded in the third region B3 of the sensor unit 640 to sense thesignal. The electronic device 100 may control the light emitter 630 suchthat the magnitude of the pulse of the light 710 output from the lightemitter 630 is 2, the period of the pulse of the light 710 is 2 ms, andthe duty cycle of the pulse of the light 710 is 20%.

The electronic device 100 according to an embodiment of the disclosuremay calculate measurement parameters (e.g., a parameter with respect toa target region, a parameter with respect to the output of an emissionlight, a parameter with respect to the sensing of a reflection light,etc.) of the depth sensor in real time based on the eye information (the2D location information of the gaze point and the estimated depthinformation) of the user obtained from the eye tracking sensor 160. Atthis time, the electronic device 100 may calculate the measurementparameters of the depth sensor using a preset equation or algorithm.

The electronic device 100 may control the light emitter 630 and thesensor unit 640 of the depth sensor by using the measurement parameterscalculated in real time.

Meanwhile, the electronic device 100 according to an embodiment of thedisclosure may perform wired or wireless communication (e.g., Wi-Fi,Bluetooth, Zigbee, infrared rays, etc.) with an external device. Theelectronic device 100 may transmit the eye information (the 2D locationinformation of the gaze point and the estimated depth information) ofthe user obtained from the eye tracking sensor 160 to the externaldevice.

For example, the matching table 750 of FIG. 7B may be stored in theexternal device connected to the electronic device 100 through wired orwireless communication. Based on the eye information and the matchingtable 750 received from the electronic device 100, the external devicemay determine the measurement parameters (e.g., the parameter withrespect to the target region, the parameter with respect to the outputof the emission light, the parameter with respect to the sensing of thereflection light, etc.) of the depth sensor.

Alternatively, the external device may calculate the measurementparameters (e.g., the parameter with respect to the target region, theparameter with respect to the output of the emission light, theparameter with respect to the sensing of the reflection light, etc.) ofthe depth sensor in real time based on the eye information received fromthe electronic device 100. At this time, the external device maycalculate the measurement parameters by using a preset equation oralgorithm.

The external device may transmit the measurement parameters of the depthsensor to the electronic device 100. The electronic device 100 may usethe received measurement parameters of the depth sensor to control thelight emitter 30 and the sensor unit 640 of the depth sensor, but theconfiguration is not limited thereto.

FIGS. 8 and 9 are reference diagrams for describing a method, performedby the electronic device 100, of determining measurement parameters whenthe depth sensor 150 uses a structured light (SL) method according to anembodiment of the disclosure.

The depth sensor 150 according to an embodiment of the disclosure mayinclude an SL depth sensor. The SL method is a method of reflectinglight of a pattern on the object 830, analyzing the shape and positionof the pattern formed on the surface of the object 830, and measuring adistance (depth information) to the object 830. In general, the SL depthsensor may project the light of a linear pattern or a dot pattern ontothe object 830. The form or pattern of the light formed on the object830 may change according to the bending of the object 830. The SL methodmay include a light projector 810 and a camera 820, and may be regardedas a structure in which one of two cameras used in a stereo image typedepth sensor is replaced with the light projector 810. In general, afilm of a fixed pattern or a liquid crystal film capable of changing apattern shape may be disposed on a path of light projected by the lightprojector 810, and the light may pass through the film, and thus theform or pattern of the light may change. For example, the SL depthsensor may analyze the shape and the position of the pattern formed bythe light projected by the light projector 810 on the surface of theobject 830 by using an algorithm to calculate depth information.

Meanwhile, the electronic device 100 according to an embodiment of thedisclosure may determine a gaze point in a whole space by using eyeinformation of a user. The electronic device 100 may obtain the eyeinformation of the user by using an eye tracking sensor and may obtain2D location information of the gaze point based on the eye informationof the user. The electronic device 100 according to an embodiment of thedisclosure may control the light projector 810 and the camera 820 basedon the 2D location information of the gaze point.

The electronic device 100 according to an embodiment of the disclosuremay control the light projector 810 to project light only to a regioncorresponding to the 2D location information of the gaze point. Forexample, the optical projector 810 may change the pattern of the liquidcrystal film to project the light to pass through a second region C2corresponding to the 2D location information of the gaze point, and maynot project light that passes through a first region, a third region,and a fourth region C1, C3, and C4. Accordingly, power consumptionaccording to driving of the light source may be reduced.

In addition, the camera 820 may also split into a plurality of regionsD1, D2, D3, and D4, and the electronic device 100 may calculate thedepth information using only image signals obtained in a regioncorresponding to the 2D location information of the gaze point among theplurality of regions D1, D2, D3, and D4.

For example, a first image 910 illustrated in FIG. 9 may be an imageshowing that light of a fixed pattern as a whole is projected onto areal space. The camera 820 may capture a pattern generated by theprojected light formed on the surface of an object. For example, asecond image 920 illustrated in FIG. 9 may be an image of a patterngenerated by light projected onto the whole real space. The electronicdevice 100 may calculate depth information about the real space byanalyzing the second image 920.

The electronic device 100 according to an embodiment of the disclosuremay determine a gaze point in the whole space by using eye informationof both eyes of a user. The electronic device 100 may obtain the eyeinformation of both eyes by using an eye tracking sensor, and may obtain2D location information of a gaze point based on the eye information ofboth eyes.

When the 2D location information of the gaze point is obtained based onthe eye information of both eyes, the electronic device 100 according toan embodiment of the disclosure may project the light of the fixedpattern onto only an ROI based on the 2D location information of thegaze point. The electronic device 100 may determine a preset region withrespect to the gaze point to which the eyes are directed as the ROI. Forexample, as shown in FIG. 9, a light projector may project the light ofthe fixed pattern onto only a rear portion 915 (an ROI) of a vehicle,and a camera may obtain a third image 930 capturing the ROI 915 ontowhich the light is projected.

Accordingly, the electronic device 100 may calculate the depthinformation about the ROI 915 without calculating depth information ofthe whole space, thereby increasing the speed of calculating the depthinformation and reducing power consumption.

FIG. 10 is a reference diagram for describing a method, performed by theelectronic device 100, of determining measurement parameters when thedepth sensor 150 uses a stereo image (SI) method according to anembodiment of the disclosure.

The depth sensor 150 according to an embodiment of the disclosure mayinclude an SI depth sensor. The SI method refers to a method ofcapturing the cubic effect of an object by using two cameras. In thiscase, the depth sensor may include two cameras. The depth sensor maycalculate depth information (distance) with respect to a specific objectbased on the principle of triangulation by using difference informationof an image viewed by each camera. The human feels the cubic effectthrough a difference between images coming into the left eye and theright eye. The depth sensor measures the distance in a manner similar tothe principle that the human eye feels the cubic effect. For example,when the depth is small (the distance is close), the difference betweenimages captured by the two cameras is large, and when the depth is large(the distance is far), the difference between the images captured by thetwo cameras is small.

In case of the SI method, because two images need be processedsimultaneously in real time, a fast processing performance of aprocessor is required and hardware processing is required. Therefore, itis difficult to process the SI method in real time using only aprocessor of a small device.

The SI depth sensor according to an embodiment of the disclosure mayinclude a first camera and a second camera. At this time, the firstcamera and the second camera may capture a real space in differentdirections at different positions. For example, the first camera maycapture the real space in a first direction at a first position toobtain a first image 1010, and the second camera may capture the realspace in a second direction at a second position to obtain a secondimage 1020. In this case, when a difference image between the firstimage 1010 and the second image 1020 is used, depth information of awhole real space may be obtained.

Meanwhile, the electronic device 100 according to an embodiment of thedisclosure may determine an ROI of the whole space by using eyeinformation of both eyes of a user. The electronic device 100 may obtainthe eye information of both eyes by using an eye tracking sensor, andmay obtain 2D location information of a gaze point based on the eyeinformation of both eyes. For example, when the gaze point of the userof the electronic device 100 is a first point, the electronic device 100may determine a preset region with respect to the first point as theROI. A first region 1015 of FIG. 10 represents a region corresponding tothe ROI in the first image 1010, and a second region 10025 represents aregion corresponding to the ROI in the second image 1020. Accordingly,the electronic device 100 may calculate depth information with respectto the ROI by calculating only the difference image between an image ofthe first region 1015 and images 1030 and 1040 of the second region1025.

When the 2D location information of the gaze point is obtained based onthe eye information of both eyes, the electronic device 100 according toan embodiment of the disclosure may determine the ROIs in the capturedfirst and second images 1010 and 1020 based on the 2D locationinformation of the gaze point and calculate the depth information withrespect to only the ROIs. Alternatively, the electronic device 100 mayenlarge and capture the ROIs by using a zoom function, obtain the images1030 and 1040 with respect to the ROIs, and calculate the depthinformation with respect to the ROI. Accordingly, the calculation speedof the depth information may increase, and the power consumption may bereduced.

FIG. 11 is a diagram for describing a method, performed by theelectronic device 100, of displaying a virtual object 1120 according toan embodiment of the disclosure.

Referring to FIG. 11, the electronic device 100 according to anembodiment of the disclosure may display the virtual object 1120 on adisplay based on obtained depth information of a gaze point. Forexample, the electronic device 100 may display the virtual object 1120in the form of augmented reality (AR). When displaying the virtualobject in the form of AR, the electronic device 100 may display thevirtual object 1120 on the display such that the virtual object 1120overlaps a real space 1110 (a 2D or 3D space of the real world) observedthrough the display. For example, the electronic device 100 may obtaindepth information of a region around the gaze point (e.g., a regionaround a desk), and give a depth similar to the obtained depthinformation to the virtual object 1120 (e.g., a vase) such that a userrecognizes the virtual object 1120 as being located in the region aroundthe desk.

FIG. 12 is a flowchart of a method of operating the electronic device100, according to an embodiment of the disclosure.

Referring to FIG. 12, the electronic device 100 according to anembodiment of the disclosure may obtain eye information of both eyes ofa user (S1210).

The electronic device 100 according to an embodiment of the disclosuremay provide light to a user's eye (the left eye and the right eye) usingan eye tracking sensor, and may sense an amount of the light reflectedfrom the user's eye. The electronic device 100 may determine the eyedirections of both eyes based on the sensed amount of light.

Alternatively, the electronic device 100 may provide light to the user'seye using the eye tracking sensor and may capture the user's eye. Inaddition, the electronic device 100 may determine the eye directions ofboth eyes based on respective images of the captured eyes.

The electronic device 100 according to an embodiment of the disclosuremay obtain a gaze point based on the eye information (S1220).

The electronic device 100 may obtain 2D coordinate information (xcoordinate value and y coordinate value) with respect to a point atwhich a user gazes based on the eye direction of the user's right eyeand the eye direction of the user's left eye. In addition, theelectronic device 100 estimate a distance (z coordinate value) to thepoint at which the user gazes based on the eye direction of the user'sright eye and the eye direction of the user's left eye. Accordingly, theelectronic device 100 may obtain 2D location information and estimateddepth information of the gaze point.

The electronic device 100 according to an embodiment of the disclosuremay determine measurement parameters of a depth sensor based oninformation about the gaze point (S1230).

The measurement parameters of the depth sensor may include at least oneof a parameter with respect to a target region, a parameter with respectto the output of an emission light (the output pattern of the emissionlight and the magnitude of the output of the emission light), or aparameter with respect to sensing of a reflection light. For example,the electronic device 100 may determine the parameter with respect tothe target region by using the 2D location information of the gazepoint, and may determine the parameter with respect to the output of theemission light (the output pattern of the emission light and themagnitude of the output of the emission light) and the parameter withrespect to sensing of the reflection light by using the estimated depthinformation of the gaze point. This is described in detail withreference to FIGS. 5A to 10, and thus a detailed description thereofwill be omitted.

The electronic device 100 according to an embodiment of the disclosuremay obtain depth information about at least one object included in apreset ROI with respect to the gaze point based on the determinedmeasurement parameter.

FIG. 13 is a diagram for describing a method, performed by theelectronic device 100, of obtaining depth information 1330 and 1340according to an embodiment of the disclosure.

A depth sensor according to an embodiment of the disclosure may includeat least one camera and obtain depth information about a real space 1310included in a field of view (FOV) of the camera included in the depthsensor. Hereinafter, a space of a range that the depth sensor is capableof sensing (the real space 1310) will be referred to as a “whole space.”

As shown in FIG. 13, the electronic device 100 according to anembodiment of the disclosure may determine an ROI 1320 of the wholespace 1310. For example, as described with reference to FIGS. 2 to 3D,the electronic device 100 may obtain the eye information of both eyes ofa user by using an eye tracking sensor and obtain a gaze point of a userbased on the eye information of both eyes. In addition, a regionpreviously set with respect to the gaze point may be determined as theROI 1320 based on the obtained gaze point. Alternatively, the electronicdevice 100 may obtain an image of the whole space 1310 and recognize amain object (e.g., a person, a face, a hand, etc.) within the determinedROI 1320 using object recognition technology, thereby determining therecognized main object as the ROI 1320 other than the region previouslyset with respect to the gaze point.

When the ROI 1320 is determined, the electronic device 100 may obtaindepth information using different measurement parameters with respect tothe ROI 1320 and the remaining space excluding the ROI 1320.

For example, the electronic device 100 may set the measurement parameterof the depth sensor as a first parameter to obtain the depth information1330 about the ROI. At this time, the electronic device 100 may set thefirst parameter based on the information of the gaze point. For example,the electronic device 100 may set the region previously set with respectto the gaze point as the ROI 1320 based on the 2D location informationof the gaze point and set a light emission region, a light sensingregion, etc. to correspond to the set ROI 1320. In addition, theelectronic device 100 may set a pattern of emitted light or light outputbased on the estimated depth information of the gaze point.

For example, when the depth sensor is a TOF depth sensor, a sensor unitmay sense signals corresponding to the ROI by increasing a sampling rateor reducing a sampling cycle and may not sense signals corresponding tothe remaining region, thereby obtaining high resolution depthinformation about the ROI.

In addition, the electronic device 100 may set the measurement parameterof the depth sensor as a second parameter to obtain the depthinformation 1340 about the remaining regions excluding the ROI 1320. Forexample, when the depth sensor is the TOF depth sensor, the sensor unitsensor unit may sense signals corresponding to the remaining region byreducing a sampling rate or increasing a sampling cycle and may notsense signals corresponding to the ROI, thereby obtaining low resolutiondepth information about the ROI.

Alternatively, the electronic device 100 may obtain low resolution depthinformation about the whole space 1310 including the ROI.

Accordingly, the electronic device 100 may obtain highly accurate depthinformation (high resolution depth information) about the ROI, and alsoobtain approximate depth information (low resolution depth information)about the remaining region (a region around the gaze point).

FIG. 14 is a flowchart illustrating a method, performed by theelectronic device 100, of obtaining depth information according to anembodiment of the disclosure.

Referring to FIG. 14, the electronic device 100 according to anembodiment of the disclosure may determine an ROI in a whole space(S1410). A method of determining the ROI is described in detail withreference to FIG. 13, and thus a detailed description thereof will beomitted.

The electronic device 100 may set measurement parameters of a depthsensor as a first parameter set to obtain depth information about thewhole space (S1420).

For example, the electronic device 100 may obtain low resolution depthinformation about the whole space.

The electronic device 100 may obtain first depth information about theROI based on the depth information about the whole space (S1430).

For example, the electronic device 100 may determine the depthinformation about the ROI included in the depth information about thewhole space as first depth information.

The electronic device 100 may determine a second parameter set based onthe first depth information (S1440). For example, the electronic device100 may determine a parameter with respect to the output of an emissionlight (the output pattern of the emission light and the magnitude of theoutput of the emission light) of a depth sensor, or a parameter withrespect to sensing of a reflection light based on the first depthinformation of the ROI.

The electronic device 100 may obtain second depth information about theROI by using the depth sensor having a measurement parameter set to thesecond parameter set (S1450). At this time, the second depth informationmay be high resolution depth information and may be depth informationhaving greater accuracy than an accuracy of the low resolution depthinformation obtained in S1420.

FIG. 15 is a diagram illustrating an example in which the electronicdevice 100 repeatedly performs operations of obtaining depth informationof FIG. 14 according to an embodiment of the disclosure.

Referring to FIG. 15, the electronic device 100 according to anembodiment of the disclosure may repeatedly perform an operation (S1420)of obtaining depth information 1510 (low resolution depth information)about a whole space and an operation (S1450) of obtaining second depthinformation 1520 (high resolution depth information) about an ROI at aregular period in an alternating fashion.

For example, the electronic device 100 may set a first period T1 forobtaining the depth information 1510 about the whole space and a secondperiod T2 for obtaining the second depth information 1520 about the ROI.At this time, the electronic device 100 may adjust the first period T1according to the movement of the electronic device 100 to adjust theupdate period of depth information of the remaining region excluding theROI. That is, the electronic device 100 may adjust the first period T1according to an amount of change in the movement of the remaining regionexcluding the ROI in an image generated by the movement of theelectronic device 100. For example, when the movement of the electronicdevice 100 is small (when the electronic device 100 is static), theelectronic device 100 may increase the first period T1. Conversely, andwhen the movement of the electronic device 100 is large (when theelectronic device 100 is dynamic), the electronic device 100 may reducea second period T2.

In addition, the electronic device 100 may adjust the second period T2according to a minimum time required for interaction between a user ofthe electronic device 100 and a virtual object displayed on a gazepoint. For example, the electronic device 100 may set the second periodT2 to be equal to or shorter than a minimum time required for updatingdepth information about a hand for interaction such as a hand gesturerecognition, but is not limited thereto.

FIG. 16 is a diagram illustrating an example in which the electronicdevice 100 provides a virtual object 1630 using an AR method accordingto an embodiment of the disclosure.

Referring to FIG. 16, the electronic device 100 according to anembodiment of the disclosure may include at least one camera (an imagesensor). For example, the at least one camera may be a depth cameraincluded in a depth sensor or a camera provided separately from thedepth sensor.

The at least one camera may obtain an image 1610 corresponding to aspace included in a FOV of the camera. The electronic device 100 maydetect a main object from the obtained image 1610. For example, theelectronic device 100 may display the virtual object 1630 using the ARmethod such that a user recognizes the virtual object 1630 as beinglocated near a real object 1620. In addition, when a user of theelectronic device 100 interacts with the virtual object 1630 by using ahand, the main object may be a user's hand 1640.

The electronic device 100 may detect a region of the hand 1640 from theobtained image 1610, determine the region of the hand 1640 as an ROI,and determine the remaining region excluding the region of the hand 1640as a background region.

The electronic device 100 may obtain high resolution depth informationabout the ROI and low resolution depth information about the backgroundregion. For example, the electronic device 100 may set the measurementparameter of the depth sensor to obtain depth information with highaccuracy with respect to the region of the hand 1640, thereby obtaininghigh resolution depth information. Meanwhile, the electronic device 100may set the measurement parameter of the depth sensor to obtain depthinformation with low accuracy with respect to the background region,thereby obtaining low resolution depth information.

The electronic device 100 may estimate a pose of the hand 1640 orrecognize a gesture of the hand 1640 using the high resolution depthinformation of the region of the hand 1640. Meanwhile, the electronicdevice 100 may perform camera pose tracking or background modeling usingthe low resolution depth information about the background region, butthe configuration is not limited thereto.

FIG. 17 is a diagram illustrating an example in which the electronicdevice 100 recognizes a face of a person using depth informationaccording to an embodiment of the disclosure.

Referring to FIG. 17, the electronic device 100 according to anembodiment of the disclosure may include at least one camera. Forexample, the at least one camera may be a depth camera included in adepth sensor or a camera provided separately from the depth sensor. Theat least one camera may obtain an image including a face.

The electronic device 100 may detect a main feature region from theobtained face image. For example, in the face of the person, eyes, nose,and mouth regions may be important regions to distinguish from otherpeople. The the electronic device 100 may detect the eyes, nose, andmouth regions from the face image. The electronic device 100 accordingto an embodiment of the disclosure may obtain high resolution depthinformation with respect to the eyes, nose, and mouth regions 1740 ofthe face and low resolution depth information with respect to theremaining region.

A first depth image 1710 of FIG. 17 represents the low resolution depthinformation obtained with respect to a whole face region, and a seconddepth image 1720 represents the high resolution depth informationobtained with respect to the whole face region. Also, a third depthimage 1730 represents high resolution depth information with respect tothe eyes, nose, and mouth regions 1740 and low resolution depthinformation with respect to the remaining region obtained by theelectronic device 100.

When the electronic device 100 performs face recognition (identityrecognition) using the third depth image 1730 according to an embodimentof the disclosure, recognition performance (recognition accuracy) may beimproved compared to when the electronic device 100 performs facerecognition using the first depth image 1710. Further, and a recognitionspeed may increase compared to when the electronic device 100 performsface recognition using the second depth image 1720. In addition, whenrecognizing a face of a person at a long distance, the electronic device100 may increase the resolution of the main feature region to obtain adepth image, thereby improving the recognition performance.

FIG. 18 is a block diagram illustrating a configuration of theelectronic device 100 according to an embodiment of the disclosure.

Referring to FIG. 18, the electronic device 100 according to anembodiment of the disclosure may include the eye tracking sensor 160,the depth sensor 150, and the processor 120.

The eye tracking sensor 160 according to an embodiment of the disclosuremay include an illuminator that provides light to a user's eye and adetector that detects light. The illuminator may include a light sourcethat provides light and a scanning mirror that controls a direction ofthe light provided from the light source. The scanning mirror maycontrol the direction to direct the light provided from the light sourcetoward the user's eye (e.g., a cornea). The detector may detect thelight reflected from the user's eye and measure an amount of thedetected light. The eye tracking sensor 160 may track the both eyes ofthe user based on the measured amount of light.

Alternatively, the eye tracking sensor 160 according to an embodiment ofthe disclosure may include the illuminator and a capturer. Theilluminator may include an infrared light emitting diode (IR LED) andprovide light (e.g., an infrared light) to the user's eye when theuser's eye is captured. Because the light is provided to the user's eye,the reflection light may be generated in the user's eye. In addition,the capturer may include at least one camera. At this time, the at leastone camera may include an infrared camera IR. The capturer may capturethe user's eye. The eye tracking sensor 160 may track the eyes of theuser based on an eye image of the user.

The depth sensor 150 according to an embodiment of the disclosure mayobtain depth information about one or more objects included in the realworld. The depth information may correspond to a distance from the depthsensor 150 to a specific object. The greater the distance from the depthsensor 150 to the specific object, the greater the depth value. Thedepth sensor 150 according to an embodiment of the disclosure may obtaindepth information of an object in various ways. For example, the depthsensor 150 may obtain the depth information using at least one of a TOFmethod, a SI method, or a SL method.

The depth sensor 150 according to an embodiment of the disclosure mayinclude at least one camera and obtain depth information about an actualspace included in a FOV of the camera included therein.

The processor 120 according to an embodiment of the disclosure maygenerally control the electronic device 100. The processor 120 accordingto an embodiment of the disclosure may execute one or more programsstored in a memory.

The memory according to an embodiment of the disclosure may storevarious data, programs or applications for driving and controlling theelectronic device 100. The program stored in the memory may include oneor more instructions. The program (one or more instructions) orapplication stored in the memory may be executed by the processor 120.

The processor 120 according to an embodiment of the disclosure mayobtain information about a gaze point based on the eye information ofthe user obtained by the eye tracking sensor 160. For example, theprocessor 120 may obtain 2D coordinate information (x coordinate valueand y coordinate value) with respect to a point at which a user gazesbased on the eye direction of the user's right eye and the eye directionof the user's left eye. In addition, the processor 120 may estimate adistance (z coordinate value) to the point at which the user gazes basedon the eye direction of the user's right eye and the eye direction ofthe user's left eye. Accordingly, the processor 120 may obtain 2Dlocation information and estimated depth information of the gaze point.

The processor 120 according to an embodiment of the disclosure may set aregion previously set with respect to the gaze point as an ROI based onthe information of the gaze point and obtain depth information about atleast one object included in the ROI. For example, the processor 120 maydetermine the measurement parameters of the depth sensor 150 based onthe information about the gaze point. The measurement parameters of thedepth sensor 150 may include a parameter with respect to a targetregion, a parameter with respect to the pattern of an emission light, aparameter with respect to the output of the emission light, etc. Forexample, the processor 120 may determine the parameter with respect tothe target region using the 2D location information of the gaze point,and determine the parameter with respect to the pattern of the emissionlight and the parameter with respect to the output of the emission lightusing the estimated depth information of the gaze point.

This is described in detail with reference to FIGS. 5 to 10, and thus adetailed description thereof will be omitted. The processor 120 mayobtain depth information about at least one object included in the ROI,based on the determined measurement parameters.

The processor 120 according to an embodiment of the disclosure may setthe measurement parameters of the depth sensor 150 as a first parameterset to obtain low resolution depth information about a whole space. Inaddition, the processor 120 may determine a second parameter set basedon the depth information about the ROI included in the low resolutiondepth information. The processor 120 may set the measurement parameterof the depth sensor 150 as the second parameter set to obtain highresolution depth information about the ROI.

FIG. 19 is a block diagram illustrating a configuration of an electronicdevice 1900 according to another embodiment of the disclosure. Theelectronic device 1900 of FIG. 19 may be an embodiment of the electronicdevice 100 of FIG. 18.

Referring to FIG. 19, the electronic device 1900 according to anembodiment of the disclosure may include a sensing unit 1910, a memory1960, a controller 1930, an outputter 1920, a user inputter 1940, and acommunicator 1950.

The controller 1930 and the memory 1960 of FIG. 19 respectivelycorrespond to the processor 120 and the memory 130 of FIG. 18, and thusthe same descriptions thereof will be omitted.

The sensing unit 1910 may sense a state of the electronic device 1900 ora state around the electronic device 1900 and may transmit sensedinformation to the controller 1930.

The sensing unit 1910 may include at least one of an image sensor 1911,a depth sensor 1912, an eye tracking sensor 1913, an acceleration sensor1914, a location sensor (e.g. a global positioning system (GPS)) 1915, atemperature/humidity sensor 1916, a magnetic sensor 1917, a gyroscopesensor 1918, or a microphone 1919, but the sensing unit 1910 is notlimited thereto.

The image sensor 1911 according to an embodiment of the disclosure mayobtain an image frame such as a still image or a moving image. Forexample, the image sensor 1911 may capture an image of the outside ofthe electronic device 1900. At this time, the image captured by theimage sensor 1911 may be processed by the controller 1930 or a separateimage processor (not shown).

The depth sensor 1912 and the eye tracking sensor 1913 of FIG. 19respectively correspond to the depth sensor 150 and the eye trackingsensor 160 of FIG. 18, and thus the same descriptions thereof will beomitted.

The microphone 1919 may receive an external sound signal and process thereceived signal as electrical speech data. For example, the microphone1919 may receive a sound signal from an external device or a speaker.The microphone 1919 may use various noise reduction algorithms foreliminating noise generated in a process of receiving an external soundsignal.

Functions of the acceleration sensor 1914, the location sensor 1915, thetemperature/humidity sensor 1916, the magnetic sensor 1917, and thegyroscope sensor 1918 will be understood by the artisan of ordinaryskill and thus, detailed descriptions thereof will be omitted.

The outputter 1920 may be an output interface to output an audio signalor a video signal or a vibration signal. The outputtter 1920 may includea display 1921, a sound outputter 1922, a vibration motor 1923, etc.

The display 1921 may display and output information processed by theelectronic device 1900. For example, the display 1921 may display avirtual object.

According to an embodiment of the disclosure, the display 1921 may be atransparent display or an opaque display. The transparent display refersto an information display device in which a backside of a screendisplaying information is reflected. The transparent display may includea transparent device, and may adjust light transmittance with respect tothe transparent device to adjust transparency or adjust an RGB value ofeach pixel to adjust transparency.

The sound outputter 1922 may output audio data received from thecommunicator 1950 or stored in the memory 1960. Also, the soundoutputter 1922 may output a sound signal related to functions (e.g.,call signal reception sound, message reception sound, and alarm sound)performed by the electronic device 1900. The sound outputter 1922 mayinclude a speaker, a buzzer, etc. According to an embodiment of thedisclosure, when an input is generated through a virtual inputinterface, the sound outputter 1922 may output an audio signalcorresponding to the generated input.

The vibration motor 1923 may output a vibration signal. For example, thevibration motor 1923 may output a vibration signal corresponding to anoutput of audio data or video data (e.g., call signal reception sound,message reception sound, etc.) Also, the vibration motor 1923 may outputthe vibration signal when an input is generated through the virtualobject.

The user inputter 1940 may be a user input interface for a user to inputdata for controlling the electronic device 1900. For example, the userinputter 1940 may include a key pad, a dome switch, a touch pad (acontact capacitance type, a pressure resistive type, an infrared raydetection type, a surface ultrasonic wave conduction type, an integraltension measurement type, a piezo effect type, etc.), a jog wheel, a jogswitch, and the like, but is not limited thereto. According to anembodiment of the disclosure, the user inputter 1940 may include avirtual input interface.

The communicator 1950 may be a communication interface that includes oneor more elements for communication between the electronic device 1900and an external device or between the electronic device 1900 and aserver. For example, the communicator 1950 may include a short-rangewireless communicator 1951, a mobile communicator 1952, and a broadcastreceiver 1953.

The short-range wireless communicator 1951 may include a Bluetoothcommunicator, a near field communicator (NFC/RFID), a WLAN (WiFi)communicator, a Zigbee communicator, an infrared data association (IrDA)communicator, an ultra wideband (UWB) communicator, an Ant+communicator, etc., but the wireless communication is not limitedthereto.

For example, the communicator 1950 may transmit eye information (2Dlocation information and estimated depth information of the gaze point)of a user obtained by the eye tracking sensor 1913 to an external deviceand may receive measurement parameters of the depth sensor 1912corresponding to the eye information of the user from the externaldevice.

The mobile communicator 1952 may transmit and receive a radio signal toand from at least one of a base station, an external terminal, or aserver on a mobile communication network. Here, the radio signal mayinclude various types of data according to a speech call signal, a videocall signal, or a text/multimedia message transmission/reception.

The broadcast receiver 1953 may receive a broadcast signal and/orbroadcast-related information from outside through a broadcast channel.The broadcast channel may include a satellite channel and a terrestrialchannel. The electronic device 1900 may not include the broadcastreceiver 1953 according to an implementation example.

The memory 1960 may store program for processing and controlling thecontroller 1930 and store input/output data (e.g., gesture informationcorresponding to an input mode, a virtual input interface, data inputthrough the virtual input interface, sensing information measured by asensor, content, etc.).

The memory 1960 according to an embodiment of the disclosure may storethe matching table 750 illustrated in FIG. 7B. Alternatively, the memory1960 may store equations, algorithms, etc. for calculating themeasurement parameters of the depth sensor 1912 based on the eyeinformation according to an embodiment of the disclosure.

For example, the matching table 750 may be stored in a read only memory(ROM). When driving the depth sensor, the controller 1930 may load thematching table 750 stored in the ROM onto a random access memory (RAM)and determine measurement parameters matching the eye information byusing the loaded matching table 750.

The controller 1930 may output the determined measurement parameters tothe depth sensor 1912 to control the depth sensor 1912 to obtain depthinformation using the determined measurement parameters.

The memory 1960 may include at least one type storage medium of a flashmemory type, a hard disk type, a multimedia card micro type, a card typememory (e.g., SD or XD memory, etc.), random access memory (RAM), staticrandom access memory (SRAM), read only memory (ROM), electricallyerasable programmable read-only memory (EEPROM), programmable read-onlymemory (PROM), a magnetic memory, a magnetic disk, or an optical disk.In addition, the electronic device 1900 may operate a web storage or acloud server on the Internet that performs a storage function of thememory 1960. The programs stored in the memory 1960 may be classifiedinto a plurality of modules depending on a function thereof, e.g., auser interface (UI) module 1961, a notification module 1962, etc.

The UI module 1961 may provide a specialized UI or graphical UI (GUI)interworking with the electronic device 1900 for each application. Inaddition, according to an embodiment of the disclosure, the UI module1961 may select and provide a virtual input interface suitable for asituation.

The notification module 1962 may generate a signal for notifying that anevent of the electronic device 1900 has occurred. Examples of the eventoccurring in the electronic device 1900 may include call signalreception, message reception, key signal input through the virtual inputinterface, schedule notification, etc. The notification module 1962 mayoutput a notification signal as a video signal through the display 1921,an audio signal through the sound outputter 1922, and/or a vibrationsignal through the vibration motor 1923. In addition, the notificationmodule 1962 may output a haptic signal using an external device.

Meanwhile, the block diagrams of the electronic devices 100 and 1900shown in FIGS. 18 and 19 are block diagrams for an embodiment of thedisclosure. Each element of the block diagrams may be integrated, added,or omitted, according to the specifications of the actual implementationof the image electronic devices 100 and 1900. That is, two or moreelements may be combined into one element, or one element may besubdivided into two or more elements when necessary. Furthermore, afunction performed in each block is for the purpose of explaining theembodiment of the disclosure, and a specific operation or device thereofdoes not limit the scope of the disclosure.

FIGS. 20 and 21 are diagrams for describing a method, performed byelectronic device 100, of automatically adjusting a focus according toan embodiment of the disclosure.

Referring to FIG. 20, when the electronic device 100 according to anembodiment of the disclosure displays a virtual object 2020 as beinglocated around a real object 2010, a user may experience avergence-accommodation conflict. For example, when a distance from theelectronic device 100 to the real object 2010 is d1, the electronicdevice 100 may display the virtual object 2020 as being located at thedistance d1. At this time, because the user sees the virtual object 2020as being located at the distance d, the vergence distance of both eyesof the user is d1. Meanwhile, because the virtual object 2020 isactually displayed on the display of the electronic device 100, thefocal distance of both eyes may be a distance d2 from the user's eyes tothe display. In this case, the vergence distance and the focal distancemay be inconsistent, and when the electronic device 100 is used for along time, the user may feel faint, dizzy, and motion sick.

Therefore, to alleviate a vergence-accommodation conflict, theelectronic device 100 according to an embodiment of the disclosure mayadjust a focal length.

Referring to FIG. 21, according to an embodiment of the disclosure, theelectronic device 100 may include a focus adjustment lens 2110. Thefocus adjustment lens 2110 may refer to an optical device capable ofadjusting optical characteristics such as a focal length or an opticalaxis position, but is not limited thereto. For example, the focusadjustment lens 2110 may locally vary the effective refractive indexaccording to the applied voltage. Liquid crystals may be generally usedfor the focus adjustment lens 2110, but the adjustment configuration isnot limited thereto.

The electronic device 100 according to an embodiment of the disclosuremay obtain eye information of a user using an eye tracking sensor, andbased on the eye information of the user, obtain information about agaze point (e.g., the real object 2010), and based on the informationabout the gaze point, obtain depth information about the gaze point.This is described in detail in FIGS. 1 to 19, and thus a descriptionthereof will be omitted.

In addition, the electronic device 100 may display the virtual object2020 based on the depth information about the real object 2010. Forexample, the electronic device 100 may display the virtual object 2020on the display such that the user recognizes the virtual object 2020 asbeing located around the real object 2010 observed through the display.

The electronic device 100 according to an embodiment of the disclosuremay adjust the focal length based on depth information of the realobject 2010 (or depth information of the virtual object 2020). Forexample, when the distance to the real object 2010 is d1, the electronicdevice 100 may adjust the focal length of the user's eye to d1 using thefocus adjustment lens 2110. At this time, the electronic device 100 mayobtain information about a first region 2121 and a second region 2122through which the user's eyes pass among the whole region of the focusadjustment lens 2110 based on the eye information of the user. Theelectronic device 100 may change the refractive index such that thefocal length of the first region 2121 and the second region 2122 is d1by adjusting the voltage applied to the focus adjustment lens 2110.Accordingly, the vergence distance and the focal distance may beconsistent, and the vergence-accommodation conflict may be prevented.

FIG. 22 is a diagram for describing a method, performed by theelectronic device 100, of performing eye based spatial modelingaccording to an embodiment of the disclosure.

Referring to FIG. 22, a whole space 2210 illustrated in FIG. 22represents a space within a range that a depth sensor included in theelectronic device 100 is capable of sensing. According to an embodimentof the disclosure, the electronic device 100 may obtain the eyeinformation of a user using an eye tracking sensor. For example, asdescribed with reference to FIGS. 2 to 4B, the electronic device 100 mayobtain 2D location information and depth information about a point or aspace gazed by the user of the electronic device 100. The electronicdevice 100 may obtain the depth information about the point or the space(the gaze point) gazed by the user based on the obtained information.This is described in detail with reference to FIGS. 1 to 19, and thus adetailed description thereof will be omitted. For example, when the usergazes at a chair 2220 in the whole space 2210, the electronic device 100may obtain depth information about the chair 2220. The electronic device100 may overlap and display a virtual object around the chair 2220 basedon depth information about the chair 2220.

In addition, when the eye of the user moves, the electronic device 100may obtain depth information about a space to which the eye of the usermoves based on information (2D location information and depthinformation) of the space to which the eye of the user has moved.

According to an embodiment of the disclosure, the electronic device 100may obtain the eye information of the user in real time, obtain a gazepoint in the whole space 2210 based on the eye information of the user,set a region previously set with respect to the obtained gaze point asan ROI, and obtain depth information about only the determined ROI. Theelectronic device 100 may perform modeling on the ROI based on theacquired depth information.

Accordingly, the electronic device 100 may perform modeling only on arequired space, thereby increasing modeling speed and reducing powerconsumption.

The method of operating the electronic device according to an embodimentof the disclosure may be implemented in the form of program commandsthat can be executed through various computer components and recorded ina computer-readable recording medium. The computer-readable recordingmedium may include a program command, a data file, a data structure andthe like solely or in a combined manner. The program command recorded inthe computer-readable recording medium may be a program commandspecially designed and configured for the embodiments of the disclosureor a program command known to be used by those of skill in the art ofthe computer software field. Examples of the computer-readable recordingmedium may include magnetic media such as a hard disk, a floppy disk,and magnetic tape, optical media such as compact disk read only memory(CD-ROM) and digital versatile disk (DVD), magneto-optical media such asa floptical disk, and a hardware device especially configured to storeand execute a program command, such as read only memory (ROM), randomaccess memory (RAM) and flash memory, etc. Further, examples of theprogram commands include machine language code created by a compiler andhigh-level language code executable by a computer using an interpreter.

Also, the electronic device and the operation method thereof, accordingto the described embodiments of the disclosure, may be included andprovided in a computer program product. The computer program product maybe traded as a product between a seller and a buyer.

The computer program product may include a software (S/W) program and acomputer-readable storage medium with a S/W program stored therein. Forexample, the computer program product may include products in the formof S/W programs (e.g., downloadable apps) distributed electronicallythrough manufacturers of electronic devices or electronic markets (e.g.,Google Play Store and App Store). For electronic distribution, at leasta portion of the S/W program may be stored in a storage medium or may begenerated temporarily. In this case, the storage medium may be a storagemedium of a server of a manufacturer, a server of an electronic market,or a relay server for temporarily storing an S/W program.

In a system including a server and a client device, the computer programproduct may include a storage medium of the server or a storage mediumof the client device. Alternatively, when there is a third device (e.g.,a smartphone) communicatively connected to the server or the clientdevice, the computer program product may include a storage medium of thethird device. Alternatively, the computer program product may includethe S/W program itself that is transmitted from the server to the clientdevice or the third device or transmitted from the third device to theclient device.

In this case, one of the server, the client device, and the third devicemay execute the computer program product to perform the method accordingto the described embodiments of the disclosure. Alternatively, two ormore of the server, the client device, and the third device may executethe computer program product to perform the method according to thedescribed embodiments of the disclosure in a distributed manner.

For example, the server (e.g., a cloud server or an Al server) mayexecute the computer program product stored in the server, to controlthe client device communicatively connected to the server to perform themethod according to the described embodiments of the disclosure.

An electronic device according to an embodiment of the disclosure mayobtain depth information based on a gaze point, and thus the efficiencyof depth sensing may increase and power consumption may decrease.

An electronic device according to an embodiment of the disclosure mayobtain depth information using a parameter optimized for a gaze point,thereby improving the accuracy of the depth information.

Although the embodiments of the disclosure have been described above indetail, the scope of the disclosure is not limited thereto and those ofordinary skill in the art will understand that various modifications andimprovements may be made therein without departing from the spirit andscope of the disclosure as defined by the following claims.

What is claimed is:
 1. A head-mounted display device comprising: an eyetracking circuitry configured to obtain a direction of a left eye of auser and a direction of a right eye of the user; a depth sensorconfigured to obtain depth information about one or more objects; andone or more processors configured to determine a gaze point based on thedirection of the left eye of the user and the direction of the right eyeof the user, and determine a measurement parameter of the depth sensorbased on the gaze point.
 2. The head-mounted display device of claim 1,wherein the one or more processors is further configured to obtaintwo-dimensional (2D) location information of the gaze point based on thedirection of the left eye of the user and the direction of the right eyeof the user.
 3. The head-mounted display device of claim 2, wherein theone or more processors is further configured to obtain estimated depthinformation of the gaze point based on the direction of the left eye ofthe user and the direction of the right eye of the user.
 4. Thehead-mounted display device of claim 1, wherein the depth sensor isfurther configured to obtain depth information about a region ofinterest (ROI) set with respect to the gaze point according to themeasurement parameter.
 5. The head-mounted display device of claim 1,wherein the measurement parameter of the depth sensor comprises at leastone of a parameter with respect to a target region, a parameter withrespect to an output of an emission light, or a parameter with respectto sensing of a reflection light.
 6. The head-mounted display device ofclaim 4, wherein the one or more processors is further configured tore-determine the measurement parameter of the depth sensor based on thedepth information about the ROI, and wherein the depth sensor is furtherconfigured to re-obtain the depth information about the ROI according tothe measurement parameter.
 7. The head-mounted display device of claim4, wherein the depth sensor is further configured to obtain the depthinformation about the ROI by using at least one of a time of flight(TOF) method, a structured light (SL) method, or a stereo image (SI)method.
 8. The head-mounted display device of claim 7, wherein, when thedepth sensor comprises a TOF depth sensor, the one or more processors isfurther configured to determine the measurement parameter based on the2D location information of the gaze point such that some light sourcescorresponding to the gaze point among light sources included in thedepth sensor are driven, and wherein the depth sensor is furtherconfigured to obtain the depth information about the ROI by driving thesome light sources.
 9. The head-mounted display device of claim 4,further comprising a display displaying a real space comprising the ROI,and wherein the one or more processors is further configured to controlthe display to display at least one virtual object on the ROI based onthe depth information about the ROI.
 10. The head-mounted display deviceof claim 4, wherein the one or more processors is further configured to:set the measurement parameter of the depth sensor to a first parameter;based on the first parameter, control the depth sensor to obtain wholedepth information about a space that the depth sensor is capable ofsensing, the space comprising the ROI; based on the whole depthinformation, obtain first depth information about the ROI; based on thefirst depth information, set the measurement parameter of the depthsensor as a second parameter; and based on the second parameter, controlthe depth sensor to obtain second depth information about the ROI. 11.An method of operating an electronic device, the method comprising:obtaining a left eye direction of a left eye of a user and a right eyedirection of a right eye of the user; obtaining a gaze point based onthe direction of the left eye of the user and the direction of the righteye of the user; and determining a measurement parameter of a depthsensor based on the gaze point.
 12. The method of claim 11, wherein theobtaining comprises obtaining two-dimensional (2D) location informationof the gaze point based on the direction of the left eye of the user andthe direction of the right eye of the user.
 13. The method of claim 12,wherein the obtaining of the information about the gaze point furthercomprises obtaining estimated depth information of the gaze point basedon the direction of the left eye of the user and the direction of theright eye of the user.
 14. The method of claim 11, further comprisingobtaining depth information about a region of interest (ROI) set withrespect to the gaze point according to the measurement parameter. 15.The method of claim 11, wherein the determining of the measurementparameter of the depth sensor comprises determining at least one of aparameter with respect to a target region, a parameter with respect toan output of an emission light, or a parameter with respect to sensingof a reflection light.
 16. The method of claim 14, further comprising:re-determining the measurement parameter of the depth sensor based onthe depth information about the ROI; and re-obtaining the depthinformation about the ROI according to the measurement parameter. 17.The method of claim 14, wherein the obtaining of the depth informationabout the ROI comprises obtaining the depth information about the ROI byusing at least one of a time of flight (TOF) method, a structured light(SL) method, or a stereo image (SI) method.
 18. The method of claim 17,wherein the determining of the measurement parameter of the depth sensorcomprises when obtaining the depth information using the TOF method,determining the measurement parameter based on the 2D locationinformation of the gaze point such that some light sources correspondingto the gaze point among light sources included in the depth sensor aredriven, and the obtaining of the depth information about the ROIcomprises obtaining the depth information about the ROI by driving thesome light sources.
 19. The method of claim 14, wherein the obtaining ofthe depth information about the ROI comprises: setting the measurementparameter of the depth sensor to a first parameter; based on the firstparameter, obtaining whole depth information about a space that thedepth sensor is capable of sensing; based on the whole depthinformation, obtaining first depth information about the ROI; and basedon the first depth information, setting the measurement parameter of thedepth sensor as a second parameter, and based on the second parameter,obtaining second depth information about the ROI.
 20. A non-transitorycomputer-readable recording medium having recorded thereon a program forexecuting the method of claim 11.